[WIP][POC] Pgaur on pair evaluation - #50791

A C++ implementation of the OnPair short-string codec (arXiv:2508.02280):
a trained dictionary of up to 16-byte tokens, greedy longest-prefix
tokenization, and a branch-free gather-copy decode that keeps per-row
random access. The dictionary budget is configurable from 9 to 16 bits;
codes are bit-packed at the dictionary's true width.

Splits each value into a shared prefix and a suffix before the symbol
table sees it, following the FSST+ thesis (Alexandre, CWI 2025). Used to
measure whether prefix extraction adds anything on top of either codec.

Measures FSST, OnPair, the prefix-extraction variants and zstd/lz4 pages
on one corpus set, on three axes: compression ratio, whole-column decode
with per-row random access, and encode. Every codec is charged a
bit-packed per-row length array so column reconstruction costs the same
across all of them. bench_common.h holds the timing and bit-packing
helpers shared with the cascade benchmark.

The Rust helper generates the corpora: TPC-H string columns, the OnPair
paper's real-world datasets, ClickBench columns, and synthetic
identifier and JSON sets.

Measures the native Parquet pages as libparquet writes them, each on its
own and followed by zstd(1) or lz4, against FSST and OnPair with and
without a generic codec on top. Also measures the dictionary-then-OnPair
cascade, which dictionary-encodes the column and OnPairs only the
distinct values.

Native pages are charged only what the writer emits, since they carry
their own lengths; the candidate codecs are charged a separate length
array. Auto-budget selection runs in per-corpus setup, outside the timed
region, as every codec's parameter choice does.

The roundtrip checker now runs every dictionary budget from 9 to 16
rather than only 16, for both the plain and the dedup layout, and exits
non-zero on any mismatch so it can gate a benchmark run. The packed
decode loop is templated on the code width and the auto budget picks a
width per column, so checking only 16 left the width the benchmarks
report unverified.

The width sweep answers whether a wider code decodes faster without the
confound a budget sweep carries. It holds one trained dictionary and its
code stream fixed and re-packs the same codes at every width up to 16,
so tokens, token count and copy width are identical and only the
unpacking differs.

At a fixed code width the (byte offset, intra-byte shift) pair a packed code
is read with repeats with a period of 8 / gcd(width, 8) codes. Unrolling the
decode loop by a whole multiple of that period makes every offset and every
shift a compile-time constant and advances the stream cursor once per group
rather than once per code.

The group size is a request that is rounded up to a whole period, so it is a
floor rather than the group emitted. That is what lets one setting be
constant-addressed at every width including 16, where the period is a single
code and unrolling by the period alone would be a no-op -- which is exactly
the high-cardinality half of the corpus set.

Measured with the builds alternating many times over, pinned, order rotated
and reversed between repetitions, every comparison a paired difference taken
inside one repetition: +4.6% on the corpus-set median with the predicated
store and +20.7% on the portable build, +34% on the two 14-bit columns where
it removes a defect in the old loop rather than raising a ceiling. Nothing
about the compressed form is touched, so no ratio moves.

The verifier gains a sweep over sixteen consecutive stream lengths per
column, which covers every remainder any group size up to sixteen can leave
after its last whole group. A column whose code count is a whole multiple of
the group never reaches the remainder path, so a whole-column check alone
cannot see a fault there: with the remainder loop deliberately made to skip
its first code, 43 of the 120 checkpoints are caught by the length sweep and
by nothing else. An empty corpus is now a failure rather than a vacuous pass.

…Pair

The reference FSST trainer cannot be widened in place. Its pair counter is a
dense square over the code space, which is a few hundred kilobytes at nine bits
and tens of gigabytes at sixteen, and its symbol type packs the bytes into a
single machine word, capping a symbol at eight bytes. Both are replaced here, the
counter by an open-addressed map over occupied pairs and the symbol by a fixed
byte array with an explicit length, so the same iterative local search can fill a
sixteen-bit table with symbols as long as OnPair's.

The trained table is handed to OnPair's own encoder through a new train-free
entry point, so the parsing pass and the decode kernel are shared and only the
table construction differs. That makes the ratio and decode columns attributable
to the table alone.

The round-trip verifier now gates rather than reports, returning non-zero on any
failure, and it decodes every FSST16 configuration twice, once through the
whole-column path and once through the bit-packed path at the width the trained
table needs, because a table of a few hundred tokens exercises a much narrower
packed loop than OnPair ever reaches.